A relation between Hebbian and MSE learning

Traditionally, adaptive learning systems are classified into two distinct paradigms---supervised and unsupervised learning. Although a lot of results have been published in these two learning paradigms, the relations between them have been seldom investigated. In this paper we focus on the relationship between the two kinds of learning and show that in a linear network the supervised learning with mean square error(MSE) criterion is equivalent to the basic anti-Hebbian learning rule when the desired signal is a zero mean random noise independent of the input. At least for this case there is a simple relationship between the two apparent different learning paradigms. INTRODUCTION: During the past three decades there has been a considerable increase of interest in adaptive learning system. Many different approaches have been proposed for the design of engineering systems which exhibit adaptation and learning capabilities. Generally speaking, most learning systems can be divided into one of two learning paradigms---supervised and unsupervised. It is accepted that the distinction between these two kinds of learning system resides on whether a teacher signal is used in learning. In learning with supervision, it is traditionally assumed that at each time instant we know in advance the desired response for the learning system, and we use the difference between the desired and the actual response to correct its behavior [Tsypkin, 1971]. In the unsupervised learning framework, an internal adaptation constraint must be specified and the system does self-learning based on this underlying rule. It is generally accepted that the supervised and the unsupervised learning are totally different learning methods. But, we believe that the way constrains are placed in the optimization is really the fundamental difference between the two learning methods. When unsupervised learning is used the output of the net is not directly constrained, but in fact an implicit input output relationship is being specified. Nadal and Parga showed that the maximum information that can be stored in the weights adapted with supervised learning is equal to the maximum information that can be transmitted by a dual network learning with the unsupervised model [Nadal and Parga 1994]. In our paper, we study the relation between Hebbian learning and MSE learning, and we show that MSE learning defaults to anti-Hebbian when the desired signal is a zero mean random noise. A RELATION BETWEEN SUPERVISED AND UNSUPERVISED LEARNING: 1. Unsupervised learning. Unsupervised learning is depicted in Figure 1. The learning goal is not specified as an output response and learning is done based on some underlying rule. The most famous unsupervised learning rule is the socalled Hebbian rule [Haykin 1994], which adapts the learning system based on its input and output data vectors where is the step size, F(.,.) is a function of both A learning system based on underlying rule output input signal Figure 1 The unsupervised system wij t ( ) ∆ ηF yi t ( )xj t ( ) ( ) = (1) η